

<!DOCTYPE html>
<html class="writer-html5" lang="en" >
<head>
  <meta charset="utf-8" />
  
  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  
  <title>Stretch Clusters &mdash; Ceph Documentation</title>
  

  
  <link rel="stylesheet" href="../../../_static/ceph.css" type="text/css" />
  <link rel="stylesheet" href="../../../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../../../_static/graphviz.css" type="text/css" />
  <link rel="stylesheet" href="../../../_static/css/custom.css" type="text/css" />

  
  
    <link rel="shortcut icon" href="../../../_static/favicon.ico"/>
  

  
  

  

  
  <!--[if lt IE 9]>
    <script src="../../../_static/js/html5shiv.min.js"></script>
  <![endif]-->
  
    
      <script type="text/javascript" id="documentation_options" data-url_root="../../../" src="../../../_static/documentation_options.js"></script>
        <script src="../../../_static/jquery.js"></script>
        <script src="../../../_static/underscore.js"></script>
        <script src="../../../_static/doctools.js"></script>
    
    <script type="text/javascript" src="../../../_static/js/theme.js"></script>

    
    <link rel="index" title="Index" href="../../../genindex/" />
    <link rel="search" title="Search" href="../../../search/" />
    <link rel="next" title="Configure Monitor Election Strategies" href="../change-mon-elections/" />
    <link rel="prev" title="手动编辑一个 CRUSH 图" href="../crush-map-edits/" /> 
</head>

<body class="wy-body-for-nav">

   
  <header class="top-bar">
    

















<div role="navigation" aria-label="breadcrumbs navigation">

  <ul class="wy-breadcrumbs">
    
      <li><a href="../../../" class="icon icon-home"></a> &raquo;</li>
        
          <li><a href="../../">Ceph 存储集群</a> &raquo;</li>
        
          <li><a href="../">集群运维</a> &raquo;</li>
        
      <li>Stretch Clusters</li>
    
    
      <li class="wy-breadcrumbs-aside">
        
          
            <a href="../../../_sources/rados/operations/stretch-mode.rst.txt" rel="nofollow"> View page source</a>
          
        
      </li>
    
  </ul>

  
  <hr/>
</div>
  </header>
  <div class="wy-grid-for-nav">
    
    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
      <div class="wy-side-scroll">
        <div class="wy-side-nav-search"  style="background: #eee" >
          

          
            <a href="../../../">
          

          
            
            <img src="../../../_static/logo.png" class="logo" alt="Logo"/>
          
          </a>

          

          
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="../../../search/" method="get">
    <input type="text" name="q" placeholder="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
  </form>
</div>

          
        </div>

        
        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
          
            
            
              
            
            
              <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../../../start/intro/">Ceph 简介</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../install/">安装 Ceph</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../cephadm/">Cephadm</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="../../">Ceph 存储集群</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="../../configuration/">配置</a></li>
<li class="toctree-l2 current"><a class="reference internal" href="../">运维</a><ul class="current">
<li class="toctree-l3"><a class="reference internal" href="../operating/">操纵集群</a></li>
<li class="toctree-l3"><a class="reference internal" href="../health-checks/">健康检查</a></li>
<li class="toctree-l3"><a class="reference internal" href="../monitoring/">监控集群</a></li>
<li class="toctree-l3"><a class="reference internal" href="../monitoring-osd-pg/">监控 OSD 和归置组</a></li>
<li class="toctree-l3"><a class="reference internal" href="../user-management/">用户管理</a></li>
<li class="toctree-l3"><a class="reference internal" href="../pg-repair/">修复 PG 不一致状态</a></li>
<li class="toctree-l3"><a class="reference internal" href="../data-placement/">数据归置概览</a></li>
<li class="toctree-l3"><a class="reference internal" href="../pools/">存储池</a></li>
<li class="toctree-l3"><a class="reference internal" href="../erasure-code/">纠删码</a></li>
<li class="toctree-l3"><a class="reference internal" href="../cache-tiering/">分级缓存</a></li>
<li class="toctree-l3"><a class="reference internal" href="../placement-groups/">归置组</a></li>
<li class="toctree-l3"><a class="reference internal" href="../balancer/">均衡器</a></li>
<li class="toctree-l3"><a class="reference internal" href="../upmap/">使用 pg-upmap</a></li>
<li class="toctree-l3"><a class="reference internal" href="../crush-map/">CRUSH 图</a></li>
<li class="toctree-l3"><a class="reference internal" href="../crush-map-edits/">手动编辑一个 CRUSH 图</a></li>
<li class="toctree-l3 current"><a class="current reference internal" href="#">Stretch Clusters</a><ul>
<li class="toctree-l4"><a class="reference internal" href="#id1">Stretch Clusters</a></li>
<li class="toctree-l4"><a class="reference internal" href="#stretch-cluster-issues">Stretch Cluster Issues</a></li>
<li class="toctree-l4"><a class="reference internal" href="#id2">Stretch Mode</a></li>
<li class="toctree-l4"><a class="reference internal" href="#stretch-mode-limitations">Stretch Mode Limitations</a></li>
<li class="toctree-l4"><a class="reference internal" href="#other-commands">Other commands</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../change-mon-elections/">Configure Monitor Election Strategies</a></li>
<li class="toctree-l3"><a class="reference internal" href="../add-or-rm-osds/">增加/删除 OSD</a></li>
<li class="toctree-l3"><a class="reference internal" href="../add-or-rm-mons/">增加/删除监视器</a></li>
<li class="toctree-l3"><a class="reference internal" href="../devices/">设备管理</a></li>
<li class="toctree-l3"><a class="reference internal" href="../bluestore-migration/">迁移到 BlueStore</a></li>
<li class="toctree-l3"><a class="reference internal" href="../control/">命令参考</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../troubleshooting/community/">Ceph 社区</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../troubleshooting/troubleshooting-mon/">监视器故障排除</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../troubleshooting/troubleshooting-osd/">OSD 故障排除</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../troubleshooting/troubleshooting-pg/">归置组排障</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../troubleshooting/log-and-debug/">日志记录和调试</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../troubleshooting/cpu-profiling/">CPU 剖析</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../troubleshooting/memory-profiling/">内存剖析</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../../man/">    手册页</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../troubleshooting/">故障排除</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../api/">APIs</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../../../cephfs/">Ceph 文件系统</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../rbd/">Ceph 块设备</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../radosgw/">Ceph 对象网关</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../mgr/">Ceph 管理器守护进程</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../mgr/dashboard/">Ceph 仪表盘</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../api/">API 文档</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../architecture/">体系结构</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../dev/developer_guide/">开发者指南</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../dev/internals/">Ceph 内幕</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../governance/">项目管理</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../foundation/">Ceph 基金会</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../ceph-volume/">ceph-volume</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../releases/general/">Ceph 版本（总目录）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../releases/">Ceph 版本（索引）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../security/">Security</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../glossary/">Ceph 术语</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../jaegertracing/">Tracing</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../translation_cn/">中文版翻译资源</a></li>
</ul>

            
          
        </div>
        
      </div>
    </nav>

    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">

      
      <nav class="wy-nav-top" aria-label="top navigation">
        
          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
          <a href="../../../">Ceph</a>
        
      </nav>


      <div class="wy-nav-content">
        
        <div class="rst-content">
        
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
            
<div id="dev-warning" class="admonition note">
  <p class="first admonition-title">Notice</p>
  <p class="last">This document is for a development version of Ceph.</p>
</div>
  <div id="docubetter" align="right" style="padding: 5px; font-weight: bold;">
    <a href="https://pad.ceph.com/p/Report_Documentation_Bugs">Report a Documentation Bug</a>
  </div>

  
  <div class="section" id="stretch-clusters">
<span id="stretch-mode"></span><h1>Stretch Clusters<a class="headerlink" href="#stretch-clusters" title="Permalink to this headline">¶</a></h1>
<div class="section" id="id1">
<h2>Stretch Clusters<a class="headerlink" href="#id1" title="Permalink to this headline">¶</a></h2>
<p>Ceph generally expects all parts of its network and overall cluster to be
equally reliable, with failures randomly distributed across the CRUSH map.
So you may lose a switch that knocks out a number of OSDs, but we expect
the remaining OSDs and monitors to route around that.</p>
<p>This is usually a good choice, but may not work well in some
stretched cluster configurations where a significant part of your cluster
is stuck behind a single network component. For instance, a single
cluster which is located in multiple data centers, and you want to
sustain the loss of a full DC.</p>
<p>There are two standard configurations we’ve seen deployed, with either
two or three data centers (or, in clouds, availability zones). With two
zones, we expect each site to hold a copy of the data, and for a third
site to have a tiebreaker monitor (this can be a VM or high-latency compared
to the main sites) to pick a winner if the network connection fails and both
DCs remain alive. For three sites, we expect a copy of the data and an equal
number of monitors in each site.</p>
<p>Note that the standard Ceph configuration will survive MANY failures of the
network or data centers and it will never compromise data consistency.  If you
bring back enough Ceph servers following a failure, it will recover. If you
lose a data center, but can still form a quorum of monitors and have all the data
available (with enough copies to satisfy pools’ <code class="docutils literal notranslate"><span class="pre">min_size</span></code>, or CRUSH rules
that will re-replicate to meet it), Ceph will maintain availability.</p>
<p>What can’t it handle?</p>
</div>
<div class="section" id="stretch-cluster-issues">
<h2>Stretch Cluster Issues<a class="headerlink" href="#stretch-cluster-issues" title="Permalink to this headline">¶</a></h2>
<p>No matter what happens, Ceph will not compromise on data integrity
and consistency. If there’s a failure in your network or a loss of nodes and
you can restore service, Ceph will return to normal functionality on its own.</p>
<p>But there are scenarios where you lose data availibility despite having
enough servers available to satisfy Ceph’s consistency and sizing constraints, or
where you may be surprised to not satisfy Ceph’s constraints.
The first important category of these failures resolve around inconsistent
networks – if there’s a netsplit, Ceph may be unable to mark OSDs down and kick
them out of the acting PG sets despite the primary being unable to replicate data.
If this happens, IO will not be permitted, because Ceph can’t satisfy its durability
guarantees.</p>
<p>The second important category of failures is when you think you have data replicated
across data centers, but the constraints aren’t sufficient to guarantee this.
For instance, you might have data centers A and B, and your CRUSH rule targets 3 copies
and places a copy in each data center with a <code class="docutils literal notranslate"><span class="pre">min_size</span></code> of 2. The PG may go active with
2 copies in site A and no copies in site B, which means that if you then lose site A you
have lost data and Ceph can’t operate on it. This situation is surprisingly difficult
to avoid with standard CRUSH rules.</p>
</div>
<div class="section" id="id2">
<h2>Stretch Mode<a class="headerlink" href="#id2" title="Permalink to this headline">¶</a></h2>
<p>The new stretch mode is designed to handle the 2-site case. Three sites are
just as susceptible to netsplit issues, but are much more tolerant of
component availability outages than 2-site clusters are.</p>
<p>To enter stretch mode, you must set the location of each monitor, matching
your CRUSH map. For instance, to place <code class="docutils literal notranslate"><span class="pre">mon.a</span></code> in your first data center</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ceph mon set_location a datacenter=site1
</pre></div>
</div>
<p>Next, generate a CRUSH rule which will place 2 copies in each data center. This
will require editing the CRUSH map directly:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ceph osd getcrushmap &gt; crush.map.bin
$ crushtool -d crush.map.bin -o crush.map.txt
</pre></div>
</div>
<p>Now edit the <code class="docutils literal notranslate"><span class="pre">crush.map.txt</span></code> file to add a new rule. Here
there is only one other rule, so this is ID 1, but you may need
to use a different rule ID. We also have two datacenter buckets
named <code class="docutils literal notranslate"><span class="pre">site1</span></code> and <code class="docutils literal notranslate"><span class="pre">site2</span></code>:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">rule</span> <span class="n">stretch_rule</span> <span class="p">{</span>
        <span class="nb">id</span> <span class="mi">1</span>
        <span class="nb">type</span> <span class="n">replicated</span>
        <span class="n">min_size</span> <span class="mi">1</span>
        <span class="n">max_size</span> <span class="mi">10</span>
        <span class="n">step</span> <span class="n">take</span> <span class="n">site1</span>
        <span class="n">step</span> <span class="n">chooseleaf</span> <span class="n">firstn</span> <span class="mi">2</span> <span class="nb">type</span> <span class="n">host</span>
        <span class="n">step</span> <span class="n">emit</span>
        <span class="n">step</span> <span class="n">take</span> <span class="n">site2</span>
        <span class="n">step</span> <span class="n">chooseleaf</span> <span class="n">firstn</span> <span class="mi">2</span> <span class="nb">type</span> <span class="n">host</span>
        <span class="n">step</span> <span class="n">emit</span>
<span class="p">}</span>
</pre></div>
</div>
<p>Finally, inject the CRUSH map to make the rule available to the cluster:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ crushtool -c crush.map.txt -o crush2.map.bin
$ ceph osd setcrushmap -i crush2.map.bin
</pre></div>
</div>
<p>If you aren’t already running your monitors in connectivity mode, do so with
the instructions in <a class="reference external" href="../change-mon-elections">Changing Monitor Elections</a>.</p>
<p>And lastly, tell the cluster to enter stretch mode. Here, <code class="docutils literal notranslate"><span class="pre">mon.e</span></code> is the
tiebreaker and we are splitting across data centers. <code class="docutils literal notranslate"><span class="pre">mon.e</span></code> should be also
set a datacenter, that will differ from <code class="docutils literal notranslate"><span class="pre">site1</span></code> and <code class="docutils literal notranslate"><span class="pre">site2</span></code>. For this
purpose you can create another datacenter bucket named <code class="docutils literal notranslate"><span class="pre">`site3</span></code> in your
CRUSH and place <code class="docutils literal notranslate"><span class="pre">mon.e</span></code> there</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ceph mon set_location e datacenter=site3
$ ceph mon enable_stretch_mode e stretch_rule datacenter
</pre></div>
</div>
<p>When stretch mode is enabled, the OSDs wlll only take PGs active when
they peer across data centers (or whatever other CRUSH bucket type
you specified), assuming both are alive. Pools will increase in size
from the default 3 to 4, expecting 2 copies in each site. OSDs will only
be allowed to connect to monitors in the same data center. New monitors
will not be allowed to join the cluster if they do not specify a location.</p>
<p>If all the OSDs and monitors from a data center become inaccessible
at once, the surviving data center will enter a degraded stretch mode. This
will issue a warning, reduce the min_size to 1, and allow
the cluster to go active with data in the single remaining site. Note that
we do not change the pool size, so you will also get warnings that the
pools are too small – but a special stretch mode flag will prevent the OSDs
from creating extra copies in the remaining data center (so it will only keep
2 copies, as before).</p>
<p>When the missing data center comes back, the cluster will enter
recovery stretch mode. This changes the warning and allows peering, but
still only requires OSDs from the data center which was up the whole time.
When all PGs are in a known state, and are neither degraded nor incomplete,
the cluster transitions back to regular stretch mode, ends the warning,
restores min_size to its starting value (2) and requires both sites to peer,
and stops requiring the always-alive site when peering (so that you can fail
over to the other site, if necessary).</p>
</div>
<div class="section" id="stretch-mode-limitations">
<h2>Stretch Mode Limitations<a class="headerlink" href="#stretch-mode-limitations" title="Permalink to this headline">¶</a></h2>
<p>As implied by the setup, stretch mode only handles 2 sites with OSDs.</p>
<p>While it is not enforced, you should run 2 monitors in each site plus
a tiebreaker, for a total of 5. This is because OSDs can only connect
to monitors in their own site when in stretch mode.</p>
<p>You cannot use erasure coded pools with stretch mode. If you try, it will
refuse, and it will not allow you to create EC pools once in stretch mode.</p>
<p>You must create your own CRUSH rule which provides 2 copies in each site, and
you must use 4 total copies with 2 in each site. If you have existing pools
with non-default size/min_size, Ceph will object when you attempt to
enable stretch mode.</p>
<p>Because it runs with <code class="docutils literal notranslate"><span class="pre">min_size</span> <span class="pre">1</span></code> when degraded, you should only use stretch
mode with all-flash OSDs.  This minimizes the time needed to recover once
connectivity is restored, and thus minimizes the potential for data loss.</p>
<p>Hopefully, future development will extend this feature to support EC pools and
running with more than 2 full sites.</p>
</div>
<div class="section" id="other-commands">
<h2>Other commands<a class="headerlink" href="#other-commands" title="Permalink to this headline">¶</a></h2>
<p>If your tiebreaker monitor fails for some reason, you can replace it. Turn on
a new monitor and run</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ceph mon set_new_tiebreaker mon.&lt;new_mon_name&gt;
</pre></div>
</div>
<p>This command will protest if the new monitor is in the same location as existing
non-tiebreaker monitors. This command WILL NOT remove the previous tiebreaker
monitor; you should do so yourself.</p>
<p>If you are writing your own tooling for deploying Ceph, you can use a new
<code class="docutils literal notranslate"><span class="pre">--set-crush-location</span></code> option when booting monitors, instead of running
<code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">mon</span> <span class="pre">set_location</span></code>. This option accepts only a single “bucket=loc” pair, eg
<code class="docutils literal notranslate"><span class="pre">ceph-mon</span> <span class="pre">--set-crush-location</span> <span class="pre">'datacenter=a'</span></code>, which must match the
bucket type you specified when running <code class="docutils literal notranslate"><span class="pre">enable_stretch_mode</span></code>.</p>
<p>When in stretch degraded mode, the cluster will go into “recovery” mode automatically
when the disconnected data center comes back. If that doesn’t work, or you want to
enable recovery mode early, you can invoke</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ceph osd force_recovery_stretch_mode --yes-i-really-mean-it
</pre></div>
</div>
<p>But this command should not be necessary; it is included to deal with
unanticipated situations.</p>
<p>When in recovery mode, the cluster should go back into normal stretch mode
when the PGs are healthy. If this doesn’t happen, or you want to force the
cross-data-center peering early and are willing to risk data downtime (or have
verified separately that all the PGs can peer, even if they aren’t fully
recovered), you can invoke</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ceph osd force_healthy_stretch_mode --yes-i-really-mean-it
</pre></div>
</div>
<p>This command should not be necessary; it is included to deal with
unanticipated situations. But you might wish to invoke it to remove
the <code class="docutils literal notranslate"><span class="pre">HEALTH_WARN</span></code> state which recovery mode generates.</p>
</div>
</div>



           </div>
           
          </div>
          <footer>
    <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
        <a href="../change-mon-elections/" class="btn btn-neutral float-right" title="Configure Monitor Election Strategies" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
        <a href="../crush-map-edits/" class="btn btn-neutral float-left" title="手动编辑一个 CRUSH 图" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
    </div>

  <hr/>

  <div role="contentinfo">
    <p>
        &#169; Copyright 2016, Ceph authors and contributors. Licensed under Creative Commons Attribution Share Alike 3.0 (CC-BY-SA-3.0).

    </p>
  </div> 

</footer>
        </div>
      </div>

    </section>

  </div>
  

  <script type="text/javascript">
      jQuery(function () {
          SphinxRtdTheme.Navigation.enable(true);
      });
  </script>

  
  
    
   

</body>
</html>